A unified probabilistic model for independent and principal component analysis

Principal component analysis (PCA) and independent component analysis (ICA) are both based on a linear model of multivariate data. They are often seen as complementary tools, PCA providing dimension reduction and ICA separating underlying components or sources. In practice, a two-stage approach is often followed, where first PCA and then ICA is applied. Here, we show how PCA and ICA can be seen as special cases of the same probabilistic generative model. In contrast to conventional ICA theory, we model the variances of the components as further parameters. Such variance parameters can be integrated out in a Bayesian framework, or estimated in a more classic framework. In both cases, we find a simple objective function whose maximization enables estimation of PCA and ICA. Specifically, maximization of the objective under Gaussian assumption performs PCA, while its maximization for whitened data, under assumption of non-Gaussianity, performs ICA.